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(54) Method of applying manipulations to a color digital image 

(57) A method for modifying a digital image having 
color values with an extended color gamut comprising 
the steps of representing the extended color gamut dig- 
ital image with a limited color gamut digital image and 
one or more associated residual image(s) representing 
a difference between the extended color gamut digital 
image and the limited color gamut digital image; speci- 
fying one or more desired image modification(s) to be 
applied to the extended color gamut digital image; and 
modifying the limited color gamut digital image and the 
residual image(s) to form a modified limited color gamut 
digital image and one or more modified residual 
image(s) in response to the specified desired image 
modification(s). 
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gamut and the color gamuts of other output devices and 
image sources represents a serious limitation on the 
usefulness of the video RGB color space. However, in 
many cases, the convenience of storing the image in a 
color space that is ready for direct display on a compu- 5 
ter video CRT has been the over-riding factor in the 
determination of the preferred color space. This has 
come at the expense of applications that can utilize the 
extended color gamut information that can have existed 
in an input image. One of the most serious limitations of 10 
this approach involves cases where it is desired to mod- 
ify the digital image. In this case, it is frequently benefi- 
cial to use the information that must be clipped off when 
representing the image in a video RGB color gamut to 
provide a better quality modified image. For example, if 75 
an image needs to be darkened due to the fact that the 
original image was over-exposed, the video RGB image 
will generally not retain the highlight detail information 
that could be used to darken the image. 
[001 1] It is an object of the present invention to over- 20 
come the limitations of the prior art by providing for the 
modification of digital images stored in a color space 
having a limited color gamut by retaining extended color 
gamut information. 

[001 2] This object is achieved in a method for modify- 25 
ing a digital image having color values with an extended 
color gamut comprising the steps of: 

a) representing the extended color gamut digital 
image with a limited color gamut digital image and 30 
one or more associated residual image(s) repre- 
senting a difference between the extended color 
gamut digital image and the limited color gamut dig- 
ital image; 

b) specifying one or more desired image modifica- 35 
tion(s) to be applied to the extended color gamut 
digital image; and 

c) modifying the limited color gamut digital image 
and the residual image(s) to form a modified limited 
color gamut digital image and one or more modified 40 
residual image(s) in response to the specified 
desired image modification(s). 

[0013] The present invention has the advantage that 
a digital image can be stored in a color space conven- 45 
ient for a particular application while overcoming the 
color gamut limitation associated with that color space. 
[0014] It has the additional advantage that the use of 
the extended color gamut information is optional. As a 
result, the benefits of the extended color gamut informa- so 
tion can be gained by applications that are able to make 
use of it, without introducing an image quality or compu- 
tation penalty for applications that do not require the 
optional information or that are not able to make use of 
it. 55 
[0015] It has the additional advantage that the 
extended color gamut information can be used to mod- 
ify the image so as to provide a larger amount of adjusti- 



bility in the image modification process. 
[0016] The image can be stored, for example, in a 
video RGB color space that is well-adapted for fast and 
convenient display on a computer system without com- 
promising the potential quality of the image. This pro- 
vides a particular advantage during the process of 
applying many types of modifications to the digital 
image. 

FIG. 1 is graph comparing the color gamuts of a 
typical video display, and a typical reflection print; 
FIG. 2 is a flowchart showing a process for making 
a limited gamut digital image in accordance with the 
present invention; and 

FIG. 3 is a flowchart showing a second process for 
making a limited gamut digital image in accordance 
with the present invention; 

FIG. 4 is a flowchart showing the reconstruction of 
a extended gamut digital image from the limited dig- 
ital image of FIG. 2; 

FIG. 5 is a flowchart showing the modification of the 
limited color gamut digital image and the residual 
image(s) in accordance with the present invention; 
FIG. 6 is a flowchart showing the modification of the 
extended color gamut digital image in accordance 
with the present invention; 

FIG. 7 is a flowchart showing the storing of informa- 
tion about desirable image modification(s) in 
accordance with the present invention. 

[0017] One method for storing a digital image in 
accordance with the present invention is shown in 
FIG.2. An extended color gamut digital image 20 has 
color values that are outside the limited color gamut of a 
storage color space. An adjust color values step 21 is 
used to limit the color values to those that will fit within 
the limited color gamut of the storage color space to 
form a limited color gamut digital image 22. Next, a rep- 
resent image in storage color space step 23 is used to 
produce a storage space digital image 24. A compute 
residual image(s) step 25 is used to determine one or 
more residual image(s) 26 representing the difference 
between the extended color gamut digital image and the 
limited color gamut digital image. The storage space 
digital image 24 and the residual image(s) 26 are then 
stored in a digital file 28 using a digital file storage step 
27. 

[001 8] A key aspect of the present invention is the cre- 
ation and use of the residual image(s) 26 representing 
the difference between the extended color gamut digital 
image and the limited color gamut digital image. Other 
prior art systems include the computation of a residual 
image, but none involve computing a difference 
between an extended color gamut digital image and a 
limited color gamut digital image. Nishihara et al. (U.S. 
Patent 4,903,317) describe the computation of a resid- 
ual image determined from the difference between an 
original image, and an image that has been com- 
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preserve color appearance as closely as possible. 
Regardless of what gamut mapping technique is used, 
there will necessarily be a loss of information and a dis- 
tortion of the color characteristics of the image. 
[0025] In many cases, the extended color gamut will s 
contain color values that have higher chroma values 
than can be represented in the limited color gamut. In 
some cases, the extended color gamut can also have a 
larger luminance dynamic range than can be repre- 
sented in the limited color gamut. In the case where it is 10 
necessary to reduce the luminance dynamic range of 
the image, one part in the implementation of the adjust 
color values step 21 is typically the application of a tone 
scale function. The tone scale function maps the input 
image intensities to output image intensities and might 15 
be applied to a luminance channel of the image, or alter- 
natively to each color channel of an RGB color repre- 
sentation. In some applications, the image being 
processed can actually be a monochrome image, e.g., a 
black-and-white image. In this case, the tonescale func- 20 
tion can be applied to the monochrome image lumi- 
nance values. 

[0026] For cases where the extended color gamut dig- 
ital image is a representation of the colors in an original 
scene, the adjust color values step 21 will typically 25 
involve determining reproduced color values that will 
produce desired aim colors on a target output device. 
For example, optimal color reproduction aims can be 
applied to determine desired video RGB aim colors for 
the original scene colors. The process of transforming 30 
the original scene color values into aim reproduced 
color values is sometimes referred to as "rendering" the 
image. 

[0027] Once the limited color gamut digital image 22 
has been determined, the next step is to represent it in 35 
the storage color space using the represent image in 
storage color space step 23. The output of this step is a 
storage space digital image 24. This step typically 
involves applying a device model, or a color space con- 
version, to determine the storage space color values 40 
that correspond to the adjusted color values of the lim- 
ited color gamut digital image 22. For example, if the 
adjusted color values were specified in terms of the 
CIELAB color space, a video display device model can 
be used to determine the corresponding video RGB val- 45 
ues that would be necessary to produce the specified 
adjusted color values. 

[0028] A compute residual image(s) step 25 is used to 
determine one or more residual image(s) 26 represent- 
ing the difference between the extended color gamut so 
digital image 20 and the limited color gamut digital 
image 22. In its simplest form, a single residual image 
26 can be calculated by simply subtracting the adjusted 
color values of the limited color gamut digital image 22 
from the input color values of the extended color gamut 55 
digital image 20. The residual image would then be in 
terms ol the color space used to represent those color 
values. Alternatively, the color values can be trans- 



formed into some other space that would be useful for 
computing the residual image. For example, it might be 
desirable to compute the residual image in a color 
space that is well-suited for compressing the residual 
image or that is convenient for use in reconstructing the 
extended color gamut digital image. 
[0029] There are several reasons why it may be 
advantageous to store multiple residual images instead 
of just a single residual image. For example, it might be 
desirable to store residual errors associated with lumi- 
nance errors in one residual image, and residual errors 
associated with chrominance errors in additional resid- 
ual images. This would enable an application to make a 
choice about which types of residual errors it would use 
during the process of determining a reconstructed 
extended color gamut digital image. 
[0030] In another case, a set of multiple residual 
images can correspond to different subsets of extended 
dynamic range image data. For example, a first residual 
image can extend the dynamic range of the digital 
image some fixed amount beyond the dynamic range 
associated with the limited color gamut digital image. A 
second residual image can then extend the dynamic 
range an additional increment beyond the extended 
dynamic range associated with the first residual image. 
In this way. an application using the extended color 
gamut digital image can use only the residual image(s) 
associated with the amount of extended dynamic range 
required by the application. 

[0031] Another reason that using multiple residual 
images is useful is for cases where the residual images 
are stored in tags in the digital file having a limited size. 
In this case, the residual image data can be broken into 
smaller pieces that would fit within the size limitations. 
For example, residual images can be determined for 
subsets of pixels in the extended color gamut digital 
image. In this way, the residual image data can be 
stored in a tiled fashion. 

[0032] Generally, the extended color gamut digital 
image and the limited color gamut digital image 22 
should be represented in the same color space before 
the residual image(s) are calculated so that the in- 
gamut colors will be given by zero residual errors. Since 
most images will only have a small fraction of color val- 
ues that are out of gamut, the residual image(s) will be 
dominated by zeros, and therefore will be highly com- 
pressible. 

[0033] For the remainder of this disclosure, the case 
of a single residual image will be described. However, it 
should be recognized that the method can be easily 
generalized to use a set of multiple residual images. 
[0034] For cases where the adjust color values step 
21 involves the application of a transform that modifies 
the color values for the colors within the limited color 
gamut as well as those outside the limited color gamut, 
the residual image is determined by directly computing 
the difference between the input color values of the 
extended color gamut digital image and the adjusted 
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gamut digital image is generally well suited for display 
on a target output device such as a video display. One 
advantage of this approach is that systems that cannot 
make use of the residual image will be able to display 
and manipulate this image directly with no image quality 
or computation disadvantage relative to the prior art 
where only the limited color gamut digital image is 
stored. However, the information that normally would 
have been discarded has now been stored in the resid- 
ual image and is available for use by systems that can 
utilize it. In this case, the limited color gamut digital 
image is extracted and the residual image from the dig- 
ital file is used to form a reconstructed extended color 
gamut digital image. 

[0039] FIG. 4 shows an example of reconstructing an 
extended color gamut digital image from the limited 
color gamut digital image and the residual image. The 
input to this process is an extended color gamut digital 
file 40 containing a limited color gamut digital image and 
a residual image created as described above. An 
extract data from digital file step 41 is used to extract the 
limited color gamut digital image 42 and the residual 
image 43. A reconstruct extended color gamut digital 
image step 44 is then used to form a reconstructed 
extended color gamut digital image 45 by combining the 
limited color gamut digital image 42 and the residual 
image 43. Typically the reconstruct extended color 
gamut digital image step 44 will involve combining the 
limited color gamut digital image 42 and the residual 
image 43. 

[0040] The reconstructed extended color gamut digital 
image can be used for many different purposes. For 
example, it can be used to form a digital image appropri- 
ate for display on an output device having a color gamut 
different from the limited color gamut of the limited color 
gamut digital image 42 in the digital file 40. This enables 
the generation of an optimal print from the original 
extended color gamut digital image, rather than a print 
limited by constraints of the storage color space. 
[0041 ] The information in the extended color gamut is 
particularly useful during the process of applying a 
modification to the digital image. Consider FIG. 5 which 
shows one method for modifying an extended color 
gamut digital image in accordance with the present 
invention. The input to this process is an extended color 
gamut digital file 40 containing a limited color gamut dig- 
ital image and a residual image created as described 
above. As in FIG. 4, an extract data from digital file step 
41 is used to extract the limited color gamut digital 
image 42 and the residual image 43. One or more 
desired image modification(s) 50 are then specified to 
be used to modify the image. The limited color gamut 
digital image 42 is then modified using a modify limited 
color gamut digital image step 51 in response to the 
desired image modif ication(s) 50 to form a modified lim- 
ited color gamut digital image 53. Likewise, the residual 
image 43 is modified using a modify residual image step 
52 in response to the desired image modif ication(s) 50 



to form a modified residual image 54. The modified lim- 
ited color gamut digital image 53 and the modified resid- 
ual image 54 taken together represent a modified 
extended color gamut digital image which can be stored 

5 in a modified digital file 55. Alternatively, the modified 
digital image can be used to produce an image on a dig- 
ital printer, or some other form of image display device. 
The modified digital image can also be transmitted to a 
remote location for storage or printing at that location. 

ro Methods for transmitting the modified digital image can 
include the use of a computer network connection, or a 
modem connected to a telephone line. 
[0042] There are many different types of desired 
image modif ication(s) 50 that can be applied to the dig- 

15 ital image. For example, there are many kinds of color 
and tone reproduction modifications that can be applied 
to an image. In some cases it will be desirable to adjust 
the color balance of the image. In other cases it will be 
desirable to adjust the density of the image so as to pro- 

20 duce a darker image or a lighter image. Other types of 
color and tone reproduction modifications can include 
changes in the contrast, hue or colorfulness of the 
image. In some cases, it will also be desirable to modify 
the color and tone reproduction characteristics to opti- 

25 mize them for a specific output device. The present 
invention is particularly well -suited to making modifica- 
tions to the color and tone reproduction characteristics 
of the image due to the fact that the extended color 
gamut information stored in the residual image will ena- 

30 ble larger changes to be made without any degradation 
in quality. For example, consider the case where an 
original image is determined to be over-exposed. In this 
case, highlight information in the limited color gamut 
digital image would have been clipped off during the 

35 process of rendering the image to the limited color 
gamut. However, this highlight information would be 
retained in the residual image. If it were desired to 
darken the image to correct for the over-exposure error, 
the information in the residual can then be used to 

40 determine a modified digital image that recovers the 
highlight detail. 

[0043] Other types of image modifications that can be 
applied to a digital image include spatial image modifi- 
cations. Examples of spatial image modifications 

45 include zooming, cropping, noise reduction, and sharp- 
ening the image. Zooming an image involves a resizing 
of the image. Cropping an image involves selecting a 
subset of an image. A noise reduction process generally 
involves reducing the appearance of image grain or 

so noise by smoothing the image in flat areas of the image. 
Sharpening an image typically involves applying a spa- 
tial convolution to the image to increase the apparent 
sharpness of edges in the image. There are many other 
types of image modifications such as those found in 

55 common image editing software programs such as the 
widely used Adobe PhotoShop. 
[0044] In some cases a single image modification 
could be desired, whereas in other cases, it may be 
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or machine readable bar code; solid state electronic Claims 
storage devices such as random access memory 

(RAM), or read only memory (ROM); or any other phys- 1. a method for representing a digital image with an 
ical device or medium employed to store a computer extended color gamut in a storage color space hav- 



color gamut digital image to fit within the limited 

[0051 ] color gamut to form a limited color gamut digital 

10 image; 

10 video RGB color gamut b) representing the limited color gamut digital 

12 reflection print color gamut image in the storage color space; 

20 extended color gamut digital image c) determining one of more residual images 

2 1 adjust color values representing differences between the extended 

22 limited color gamut digital image is color gamut digital image and the limited color 

23 represent image in storage color space gamut digital image; and 

24 storage space digital image d) associating the one of or more residual 

25 compute residual image image(s) with the limited color gamut digital 

26 residual image image in the storage color space such that the 

27 digital file storage 20 associated set of residual images and the lim- 

28 digital file ited color gamut digital image is adapted to be 

30 extended color gamut digital image used to form a reconstructed extended color 

31 adjust color values gamut digital image. 

32 represent image in storage color space 

33 limited color gamut digital image 25 2. The method of claim 1 further including the step of 

34 represent image in extended storage color using one or more residual images from the set of 
space residual images together with the limited color 

35 compute residual image gamut digital image to form a reconstructed 

36 residual image extended color gamut digital image. 

37 digital file storage 30 

38 digital file 3. A method for representing a digital image having 

40 digital file color values with an extended color gamut in a stor- 

41 extract data from digital file age color space having a limited color gamut com- 

42 limited color gamut digital image prising the steps of: 

43 residual image 35 

44 reconstruct extended color gamut digital image a) adjusting the color values of the extended 

45 reconstructed extended color gamut digital color gamut digital image to fit within the limited 
image color gamut to form a limited color gamut digital 

50 desired image modification(s) image; 

51 modify limited color gamut digital image 40 b) representing the limited color gamut digital 

52 modify residual image image in the storage color space; 

53 modified limited color gamut digital image c) compressing the limited color gamut digital 

54 modified residual image image; 

55 modified digital file d) determining a set of residual images repre- 

60 modify extended color gamut digital image 45 senting a difference between the extended 

61 desired image modification(s) color gamut digital image and an uncom- 

62 modified extended color gamut digital image pressed version of the compressed limited 

63 modified limited color gamut digital image color gamut digital image; and 

64 modified residual image e) associating the set of residual images with 

65 modified digital file so the limited color gamut digital image in the stor- 

70 desired image modif ication(s) age color space such that the associated resid- 

71 modify limited color gamut digital image ual images and the limited color gamut digital 

72 modified limited color gamut digital image image is adapted to be used to form a recon- 

73 preview modified image structed extended color gamut digital image. 

74 information about modification(s) ss 

75 modified digital file 4. A method for modifying a digital image having color 

values with an extended color gamut comprising 
the steps of: 
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(54) Method of applying manipulations to a color digital image 

(57) A method for modifying a digital image having 
color values with an extended color gamut comprising 
the steps of representing the extended color gamut dig- 
ital image with a limited color gamut digital image and 
one or more associated residual image(s) representing j — 
a difference between the extended color gamut digital 
image and the limited color gamut digital image; speci- 
fying one or more desired image modification (s) to be 
applied to the extended color gamut digital image; and 
modifying the limited color gamut digital image and the 
residual image(s) to form a modified limited color gamut 
digital image and one or more modified residual 
image(s) in response to the specified desired image 
modification(s). 
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